8386591: C2: wrong result because of broken truncation check in CountedLoopConverter::TruncatedIncrement::build - #31502

The third bug covered by #31501.

As we have seen in #31395, the CountedLoopConverter::has_truncation_wrap algorithm had some peculiarities, specifically this one:

For char, we recognize & 0x7fff, and not the actual char truncation (& 0xffff). So we don't actually turn (char) truncation into CountedLoops. Actually, this is a bug, see: JDK-8386591.

So here we are. The issue is that we recognized 0x7fff as char truncation, and accordingly only check that we cannot overflow/wrap the range CHAR = [0 .. 0xffff], so if we got a value in range 0x8000 .. 0xffff, that ends up overflowing/wrapping, but since we only checked for overflow outside the CHAR range, we wrongly decide there is no wrap/overflow.

Fix for 0x7fff:

I now allow both masks 0x7fff and 0xffff, for ranges 0..0x7fff and 0..0xffff respectively. I have no idea why we ever had 0x7fff in the first place, because 0xffff is what's used for char cast, and I don't know of any Java language truncation that would produce the 15-bit unsigned truncation of 0x7fff. But still: we used to optimize it, and so I'm keeping it to avoid regressions.

Also addressing signed 8-bit truncation

I also decided to address an issue in the signed truncation at the same time. I suppose I could split that out into an RFE, but it also seems like a similar oversight, also mentioned in #31395:

suspicious: shift == 8 is ((x << 8) >> 8) is signed 24-bit truncation. But the implementation maps to BYTE. This looks like an original "bug", which does not create any real issues, just means we miss optimizations for byte. We should probably match shift == 24 instead, which would be <<24 >>24.

So I slightly fixed/refactored that part too.

Testing

• I added 3 regression scenarios for the 0x7fff wrap issue.
• I was able to turn some IR rules positive in TestHasTruncationWrap.java.

This bug was found by the fuzzer from #31501, which I will integrate soon after this issue is fixed, and then we'll have even better coverage going forward.

• I confirm that I make this contribution in accordance with the OpenJDK Interim AI Policy.

Progress

• Change must not contain extraneous whitespace
• Commit message must refer to an issue
• Change must be properly reviewed (2 reviews required, with at least 1 Reviewer, 1 Author)

Issue

• JDK-8386591: C2: wrong result because of broken truncation check in CountedLoopConverter::TruncatedIncrement::build (Bug - P3) ⚠️ Issue is already resolved. Consider making this a "backport pull request" by setting the PR title to Backport <hash> with the hash of the original commit. See Backports.

Reviewing